US Politics Part II

Christopher Marks

2019-01-18

PART II: Update Data and Analyze

This is the second part of a two-part vignette designed to collect and analyze Twitter posts from members of the US Congress and the US President. This part of the vignette carries out the database initialization and initial status collections. This part of the vignette consists of two sub-parts: (a) updating the twitter database with new posts, and (b) analysis. These two sub-parts can be run together or independently, as often as required.

Very Important Note: As with the first part of this vignette, Twitter data collection can take a very long time. In this part of the vignette, collection is carried out using the update_user_timelines function. If the Twitter database has been kept up to date, i.e., if this function has been called recently on the Twitter data connection, then there will not be many new statuses and the method will run in a few minutes. However, if this function has not been called for a longer period of time, and the database consists of all members of congress, then there will be more new statuses to collect and the function will take longer to complete.

Initialization

We begin by allocating additional memory for RJava to support sentiment analysis, loading the twinfoR namespace, establishing the Twitter database connection to the sqlite database created in Part I of this vignette, and loading the authentication vector. See (and execute) the us_politics_1 vignette if you haven’t already done so.

This code assumes the auth.vector from Part I is saved. If it isn’t, reauthenticate using the authorize_IT() function as done in Part I.

options(java.parameters = "-Xmx1024m")
library(twinfoR)
setwd("~/USPol")
con <- twitter_database("USPol.sqlite")
load("auth_vector.RData") 

Update Database with New Statuses

This function should take considerably less time to run than the initial call, because it only collects new statuses. Again, we divert output to the NULL device.

# sink("/dev/null") # Recommended to suppress Rsentiment output, but
## might not work on all platforms.
update_user_timelines(
  con,
  calc.Rsentiment = TRUE,
  calc.syu = TRUE
)
# sink()
# Summarize database
summarize_database(con)
#> Query users: 558
#> User categories:
#> party
#> body
#> since_id
#> friends_collected
#> followers_collected
#> No query text table.
#> Users: 579774
#> Statuses: 1599290
#> Unique user mentions: 123710
#> Unique hashtags: 81459
#> Unique URLs: 521248
#> Total Images: 532138
#> Unique Images: 497016
#> Total relationships: 1095918
#> Database file size: 1390MB

Analysis

This section goes quickly through the analysis capability available in the twinfoR package.

Top hashtags from democrats and republicans in the last week

Top Hashtags Used by Democrats and Republicans
Dem.Hashtag Count
#TrumpShutdown 819
#EndTheShutdown 410
#ShutdownStories 109
#RaiseTheWage 102
#ForThePeople 81
#Shutdown 46
#EndTheShutdownNow 32
#2020Census 29
#WheresMitch 25
#StopWheeler 24
Rep.Hashtag Count
#SecureOurBorder 58
#Senate 25
#bordersecurity 23
#whywemarch 23
#NRDemsFakeHearing 21
#MarchForLife 16
#ProLife 15
#HumanTraffickingAwarenessDay 14
#SOTU 14
#SchoolChoice 13

Top usermentions from democrats and republicans in the last week

Democrat User Mentions
Screen_Name Name Mentions
@realDonaldTrump Donald J. Trump 281
@senatemajldr Leader McConnell 171
@HouseDemocrats House Democrats 102
@POTUS President Trump 82
@EPA U.S. EPA 81
@AppropsDems House Appropriations 67
@SpeakerPelosi Nancy Pelosi 63
@SenateGOP Senate Republicans 58
@BobbyScott Rep. Bobby Scott 53
@SenateDems Senate Democrats 39
Republican User Mentions
Screen_Name Name Mentions
@realDonaldTrump Donald J. Trump 138
@POTUS President Trump 88
@HouseGOP House Republicans 31
@ChuckGrassley ChuckGrassley 26
@WhiteHouse The White House 26
@SpeakerPelosi Nancy Pelosi 25
@senatemajldr Leader McConnell 24
@RepSamGraves Rep. Sam Graves 24
@JohnCornyn Senator John Cornyn 21
@SenTedCruz Senator Ted Cruz 19

Top URLs posted by democrats and republicans in the last week

Top Democrat and Republican URLs
Dem.URL Count
https://www.nytimes.com/interactive/2019/01/14/us/politics/women-of-the-116th-congress.html 29
https://twitter.com/BobbyScott/status/1085510013903388673 12
https://www.politico.com/story/2019/01/17/report-trump-administration-migrants-1093242 11
https://twitter.com/RepHastingsFL/status/1084874860084559878 9
https://twitter.com/ComdtUSCG/status/1085246326944788482 8
https://twitter.com/ap/status/1085192422395539457 7
https://twitter.com/speakerpelosi/status/1085549944969285634 7
http://CoveredCA.com 6
http://www.coveredca.com 6
https://trumpshutdownstories.us/ 6
Rep.URL Count
https://bit.ly/2T3fQdK 8
https://bit.ly/2D3l0Rl 6
https://bit.ly/2SNEIFX 6
https://twitter.com/RepHastingsFL/status/1084874860084559878 5
https://wcfcourier.com/opinion/columnists/guest_columnists/grassley-top-priority-is-reduce-health-care-costs/article_68335073-8c94-5072-ae42-fa5f049d0747.html 5
https://qctimes.com/opinion/columnists/building-on-the-success-of-tax-reform/article_4c9ef941-9198-5149-9537-456d72326a8a.html 4
https://siouxcityjournal.com/opinion/other-voices-grassley-free-fair-trade-will-continue-to-be/article_84579748-fa2b-53c3-9e22-5f71f027eade.html 4
https://www.foxbusiness.com/economy/us-veteran-unemployment-rate-hits-all-time-low-in-2018 4
https://www.ronjohnson.senate.gov/public/index.cfm/telephone-town-hall-signup 4
https://bit.ly/2RT2aV8 3

Top media from democrats and republicans in the last week

Top Media Posted by Democrats and Republicans
Dem.image_url count
http://pbs.twimg.com/ext_tw_video_thumb/1083742190231261186/pu/img/7V0fWabZRxGQgyDu.jpg 6
http://pbs.twimg.com/media/DwFJrATWoAAjZw2.jpg 5
http://pbs.twimg.com/media/DwFZC2zWkAEACXf.jpg 4
http://pbs.twimg.com/media/DwaZspyWsAEOnnZ.jpg 4
http://pbs.twimg.com/amplify_video_thumb/1083376535829127169/img/HNTJBuJVdoB_j_Fz.jpg 4
http://pbs.twimg.com/media/DwpD_K7X4AA-MKy.jpg 4
http://pbs.twimg.com/media/DwpEBnwX4AAiuJW.jpg 4
http://pbs.twimg.com/media/DwCyawfWoAA-IMz.jpg 3
http://pbs.twimg.com/amplify_video_thumb/1082692955075416066/img/fWdPWlMvb9nGkl4x.jpg 3
http://pbs.twimg.com/amplify_video_thumb/1083169291548139520/img/7MQJrCyTS30s6cLH.jpg 3
Rep.image_url count
http://pbs.twimg.com/amplify_video_thumb/1083413887427506177/img/wXRHJp_cZmVfXK4j.jpg 19
http://pbs.twimg.com/amplify_video_thumb/1085262852645109762/img/2mxXW9TDXH_tBgTM.jpg 8
http://pbs.twimg.com/media/DwF-sJCWsAgz3cx.jpg 5
http://pbs.twimg.com/ext_tw_video_thumb/1083493653161467905/pu/img/IPJX8VlLicdLy6yF.jpg 5
http://pbs.twimg.com/ext_tw_video_thumb/1083554750971502594/pu/img/V0LLiT7QdWwlLekH.jpg 5
http://pbs.twimg.com/media/DwAzVExW0AI3ZD4.jpg 4
http://pbs.twimg.com/amplify_video_thumb/1081227069172916224/img/G4EkQHGMBmryVXvB.jpg 3
http://pbs.twimg.com/media/Dwe-NhUW0AAHftU.jpg 3
http://pbs.twimg.com/media/Dwe-NiIX0AAMT1j.jpg 3
http://pbs.twimg.com/media/Dwe-NvFXgAEAsg5.jpg 3
Dem Image Rep Image
Top Democrat Image Top Republican Image

The top_media and status_media have many options for saving and rendering images. See documentation for details.

Top tweeters democrats and republicans in the last week

Top Democrat Tweeters
Name Location Description Tweets
Rep. Veronica Escobar El Paso, TX Tejana, mother & public servant. – Proudly representing Texas’ 16th Congressional District in the U.S. House of Representatives. 132
Rep. Don Beyer Northern Virginia & DC Representing Northern Virginia’s #VA8 in Congress. Vice-Ranking @SciCmteDems, top Science Oversight Dem. Co-Chair @CaucusOnClimate. #StopGunViolence #EqualPay 132
EPW Democrats News from the minority office of the U.S. Senate Committee on Environment and Public Works, led by Ranking Member @SenatorCarper 112
Senator Patty Murray WA & DC Official account of U.S. Senator Patty Murray (D-WA) | Tweets come from staff unless signed “-PM” by Senator Murray | RT≠endorsement 108
Rep. Pramila Jayapal Seattle, WA Congresswoman & lifelong organizer. Proudly serving WA-07. Co-Chair of @USProgressives, Senior Whip, Budget Vice Ranking. https://t.co/d22w0Bpl8R 102
Sheldon Whitehouse Rhode Island U.S. Senator from #RhodeIsland, the Ocean State. RT≠endorsement. #TimeToWakeUp 99
Rep. Joe Neguse Lafayette, Colorado Representative for Colorado’s 2nd Congressional District. Official Account. 88
Senator Cortez Masto U.S. Senator from the great state of Nevada. 80
Rep. Val Demings Washington, DC I Orlando, FL U.S. Representative Florida’s 10th Congressional District. Public Servant. Former Orlando Police Chief. Christian. Wife. Mother. Grandmother. Harley Enthusiast. 78
Congresswoman Mary Gay Scanlon Congresswoman from Pennsylvania’s 7th Congressional District. 74
Top Republican Tweeters
Name Location Description Tweets
Senator John Cornyn Austin, Texas Personally curated news I find interesting, pics, and opinions. Honored to represent the great state of Texas in the US Senate. RTs =/= endorsements 94
Rep. Bradley Byrne Southwest Alabama I am proud to represent Alabama’s First Congressional District in the U.S. House of Representatives. Official Congressional Account 78
Senator Ted Cruz Houston, Texas Representing the State of Texas in the United States Senate. 74
President Trump Washington, D.C. 45th President of the United States of America, @realDonaldTrump. Tweets archived: https://t.co/eVVzoBb3Zr 55
Rob Portman Ohio United States Senator from the Buckeye State. 52
Senator Rubio Press Florida, USA Official account of U.S. Senator @MarcoRubio’s PressShop. For more information please visit his website. 46
Senator Jerry Moran Kansas Official account of U.S. Senator Jerry Moran. Honored to represent Kansans in the U.S. Senate. 39
Sen. Lamar Alexander Maryville, TN 7th generation Tennessean born and raised in Maryville #LowerHealthCareCosts #OpioidCrisisResponse #MusicModernizationAct #RestoreOurParksAct 38
Ralph Norman Rock Hill, SC Official account of Congressman Ralph Norman. Proudly representing the 5th District of South Carolina. 37
Sean Duffy Devoted husband of one, proud father of eight, and the Representative for Wisconsin’s Seventh District. Also at https://t.co/Y7pV5IPxzR. #GoPackGo 36

Most liked tweets democrats and republicans in the last week

Most Liked Democrat Tweets
Name Created Likes Text
Maxine Waters 2019-01-14 03:51:11 93297 Trump, you didn’t get $5 billion to carry out your ill advised election promise & now you want to declare an emergency. The only emergency facing our country is the need to impeach you before you destroy our country, our ideals, & leadership in the world. Trump, go home. Bye bye!
Adam Schiff 2019-01-13 18:47:19 77350 GOP controlled the House and Senate for two years under Trump. If the wall was such an emergency, they could have used reconciliation to pass it with 50 votes in Senate. Why didn’t they? – – I guess throwing millions off health care & cutting corporate taxes were higher priorities.
Rep. Eric Swalwell 2019-01-11 00:53:11 72939 Gun violence is a national emergency
Adam Schiff 2019-01-16 01:42:24 72657 Call me old fashioned, but America deserves an Attorney General committed to following the advice of ethics lawyers. – – Call me naive, but there are attorneys qualified to serve as AG who have not expressed hostility to the Mueller probe. – – Call me what you will, but demand better.
Rep. Eric Swalwell 2019-01-15 04:58:20 69753 I don’t give two french fries about what @realDonaldTrump serves at the @WhiteHouse. But we should all care about what food is on the tables of America’s 800,000 federal workers who aren’t being paid right now. #TrumpShutdown
Nancy Pelosi 2019-01-16 14:50:50 69133 Today, I wrote to @realDonaldTrump recommending that we delay the State of the Union until after government re-opens, as the @SecretService, the lead federal agency for #SOTU security, faces its 26th day without funding. https://t.co/K2oL8WGvqo https://t.co/g3fIlxDbbK
Rep. Eric Swalwell 2019-01-13 21:56:00 62784 Six months ago, @RepAdamSchiff and I tried to subpoena the interpreter from the Trump-Putin meeting. The GOP blocked us. We knew then something was fishy. We now know Trump took the notes. Lost time and more damage to our democracy is the cost of GOP obstruction. https://t.co/o6eSXeb6vf
Rep. Eric Swalwell 2019-01-13 00:02:19 60179 Destruction of evidence is consciousness of guilt. At this point please show me evidence that @realDonaldTrump is not working for Russia. https://t.co/t3DOGmNVv8
Adam Schiff 2019-01-13 15:02:57 56825 Last year, we sought to obtain the interpreter’s notes or testimony, from the private meeting between Trump and Putin. The Republicans on our committee voted us down. Will they join us now? Shouldn’t we find out whether our president is really putting “America first?” https://t.co/hzgfTRl653
Mark Warner 2019-01-11 01:39:56 55784 I think we’ve had enough of the President running the government like one of his businesses.
Most Liked Republican Tweets
Name Created Likes Text
Senator Rand Paul 2019-01-17 13:30:26 65178 Senator McConnell is in charge of the Senate. If Mrs. Pelosi refuses to allow the president to deliver the State of the Union in the House, I propose we move it to the Senate and make it happen!
Senator Rand Paul 2019-01-16 23:00:46 52315 I have never been prouder of President Donald Trump. In today’s meeting, he stood up for a strong America and steadfastly opposed foreign wars. Putting America First means declaring victory in Afghanistan and Syria. President Trump is delivering on his promises.
Senator Rand Paul 2019-01-15 22:18:54 51471 I agree with @realDonaldTrump! It’s time to declare victory in Afghanistan and bring our troops home. We spend $46 BILLION a year in Afghanistan! How about we take that money and build roads and bridges and strengthen border security here in America? #AmericaFirst
Ron DeSantis 2019-01-11 22:04:52 49700 Effective immediately, I am officially suspending Broward County Sheriff Scott Israel for his repeated failures, incompetence and neglect of duty. https://t.co/tkHzxTHhjH
Rep. Jim Jordan 2019-01-16 13:42:59 46788 Here’s why we can’t get a deal on securing the border and ending the shutdown: today’s Left has taken the most radical positions in American history. https://t.co/UG8P5efPfb
Rep. Jim Jordan 2019-01-15 18:58:40 45207 In case you needed MORE proof that Democrats are more interested in stopping the President than helping the country… – – They rejected President Trump’s invitation to meet at the White House. They refuse to come to a compromise to secure the border and end the shutdown.
Senator Rand Paul 2019-01-16 18:10:26 42397 Sunnis have been killing Shia since the massacre at Karbala in 680 AD. If we wait until they stop killing each other, we will stay for a thousand years or more. I agree with @realDonaldTrump. Bring the troops home.
Mark Meadows 2019-01-15 17:30:34 32644 House Democrats just rejected an invitation from President Trump to come to the White House for a bipartisan discussion on border security solutions. – – And they expect voters to believe they’re serious about negotiating to find a compromise? Americans aren’t buying it.
Rep. Jim Jordan 2019-01-13 16:58:31 32205 Judiciary Chairman @RepJerryNadler: “There is no reason to doubt the seriousness or professionalism of the FBI” – – Really? No reason to doubt? – – -Andy McCabe fired for serial lying – -Lisa Page demoted/left FBI for extreme animus against @POTUS – -Peter Strzok demoted/fired, same reason
Mark Meadows 2019-01-14 14:42:46 30706 The NYT piece on the FBI investigating Trump was intended to absolve FBI leadership. It actually does the opposite. It reveals FBI chose to investigate POTUS to retaliate against him for firing Comey. They NEVER had any real evidence but opened an investigation anyway. Unreal.

Most retweeted tweets democrats and republicans in the last week

Most Retweeted Democrat Tweets
Name Created Retweet_Count Text
Adam Schiff 2019-01-13 18:47:19 27854 GOP controlled the House and Senate for two years under Trump. If the wall was such an emergency, they could have used reconciliation to pass it with 50 votes in Senate. Why didn’t they? – – I guess throwing millions off health care & cutting corporate taxes were higher priorities.
Maxine Waters 2019-01-14 03:51:11 23857 Trump, you didn’t get $5 billion to carry out your ill advised election promise & now you want to declare an emergency. The only emergency facing our country is the need to impeach you before you destroy our country, our ideals, & leadership in the world. Trump, go home. Bye bye!
Rep. Eric Swalwell 2019-01-13 21:56:00 23289 Six months ago, @RepAdamSchiff and I tried to subpoena the interpreter from the Trump-Putin meeting. The GOP blocked us. We knew then something was fishy. We now know Trump took the notes. Lost time and more damage to our democracy is the cost of GOP obstruction. https://t.co/o6eSXeb6vf
Rep. Eric Swalwell 2019-01-13 00:02:19 21085 Destruction of evidence is consciousness of guilt. At this point please show me evidence that @realDonaldTrump is not working for Russia. https://t.co/t3DOGmNVv8
Rep. Susie Lee 2019-01-17 13:27:20 19602 My fellow new Members & I were elected to work. But @SenateMajLdr McConnell seems to think otherwise. After 27 days, he still hasn’t allowed a single vote in the Senate on multiple House plans to re-open government. RT if you think he’s wrong! #EndTheShutdown #WheresMitch
Adam Schiff 2019-01-16 01:42:24 19602 Call me old fashioned, but America deserves an Attorney General committed to following the advice of ethics lawyers. – – Call me naive, but there are attorneys qualified to serve as AG who have not expressed hostility to the Mueller probe. – – Call me what you will, but demand better.
Adam Schiff 2019-01-13 15:02:57 19293 Last year, we sought to obtain the interpreter’s notes or testimony, from the private meeting between Trump and Putin. The Republicans on our committee voted us down. Will they join us now? Shouldn’t we find out whether our president is really putting “America first?” https://t.co/hzgfTRl653
Kirsten Gillibrand 2019-01-17 18:40:54 18588 .@senatemajldr has scheduled a vote today. No, it’s not to reopen the government and begin paying 800,000 federal workers who’ve gone without pay for 27 days. It’s to restrict reproductive health care coverage. Unbelievable. – – Stop attacking women. Open the government.
Nancy Pelosi 2019-01-16 14:50:50 17917 Today, I wrote to @realDonaldTrump recommending that we delay the State of the Union until after government re-opens, as the @SecretService, the lead federal agency for #SOTU security, faces its 26th day without funding. https://t.co/K2oL8WGvqo https://t.co/g3fIlxDbbK
Rep. Eric Swalwell 2019-01-11 00:53:11 16819 Gun violence is a national emergency
Most Retweeted Republican Tweets
Name Created Retweet_Count Text
Rep. Jim Jordan 2019-01-16 13:42:59 20236 Here’s why we can’t get a deal on securing the border and ending the shutdown: today’s Left has taken the most radical positions in American history. https://t.co/UG8P5efPfb
Rep. Jim Jordan 2019-01-15 18:58:40 19991 In case you needed MORE proof that Democrats are more interested in stopping the President than helping the country… – – They rejected President Trump’s invitation to meet at the White House. They refuse to come to a compromise to secure the border and end the shutdown.
Senator Rand Paul 2019-01-17 13:30:26 19784 Senator McConnell is in charge of the Senate. If Mrs. Pelosi refuses to allow the president to deliver the State of the Union in the House, I propose we move it to the Senate and make it happen!
Senator Rand Paul 2019-01-15 22:18:54 15587 I agree with @realDonaldTrump! It’s time to declare victory in Afghanistan and bring our troops home. We spend $46 BILLION a year in Afghanistan! How about we take that money and build roads and bridges and strengthen border security here in America? #AmericaFirst
Senator Rand Paul 2019-01-16 23:00:46 15231 I have never been prouder of President Donald Trump. In today’s meeting, he stood up for a strong America and steadfastly opposed foreign wars. Putting America First means declaring victory in Afghanistan and Syria. President Trump is delivering on his promises.
Mark Meadows 2019-01-15 17:30:34 14660 House Democrats just rejected an invitation from President Trump to come to the White House for a bipartisan discussion on border security solutions. – – And they expect voters to believe they’re serious about negotiating to find a compromise? Americans aren’t buying it.
Ron DeSantis 2019-01-11 22:04:52 14099 Effective immediately, I am officially suspending Broward County Sheriff Scott Israel for his repeated failures, incompetence and neglect of duty. https://t.co/tkHzxTHhjH
Senator Rand Paul 2019-01-16 18:10:26 13792 Sunnis have been killing Shia since the massacre at Karbala in 680 AD. If we wait until they stop killing each other, we will stay for a thousand years or more. I agree with @realDonaldTrump. Bring the troops home.
Rep. Jim Jordan 2019-01-13 16:58:31 13290 Judiciary Chairman @RepJerryNadler: “There is no reason to doubt the seriousness or professionalism of the FBI” – – Really? No reason to doubt? – – -Andy McCabe fired for serial lying – -Lisa Page demoted/left FBI for extreme animus against @POTUS – -Peter Strzok demoted/fired, same reason
Mark Meadows 2019-01-14 14:42:46 13252 The NYT piece on the FBI investigating Trump was intended to absolve FBI leadership. It actually does the opposite. It reveals FBI chose to investigate POTUS to retaliate against him for firing Comey. They NEVER had any real evidence but opened an investigation anyway. Unreal.

Way ahead

There are many more interesting analyses that could be done with this set. For example, do how to democrats’ and republicans’ sentiments vary when they use different hashtags or user mentions? Can we classify users as democrat or republican by their Twitter timelines? Hopefully, this package provides a useful toolset in collecting and analyzing Twitter data to answer questions like these.

Don’t forget to disconnect!